Imager For Detecting Visual Light and Projected Patterns

ABSTRACT

Methods and systems for depth sensing are provided. A system includes a first and second optical sensor each including a first plurality of photodetectors configured to capture visible light interspersed with a second plurality of photodetectors configured to capture infrared light within a particular infrared band. The system also includes a computing device configured to (i) identify first corresponding features of the environment between a first visible light image captured by the first optical sensor and a second visible light image captured by the second optical sensor; (ii) identify second corresponding features of the environment between a first infrared light image captured by the first optical sensor and a second infrared light image captured by the second optical sensor; and (iii) determine a depth estimate for at least one surface in the environment based on the first corresponding features and the second corresponding features.

CROSS-REFERENCE TO RELATED APPLICATION

The present patent application is a divisional application of U.S.patent application Ser. No. 14/672,369, filed on Mar. 30, 2015, theentire contents of which are hereby incorporated by reference.

BACKGROUND

Robotic systems, such as a robotic manipulator containing a grippingcomponent, may be used for applications involving picking up or movingobjects. For instance, a robotic device may be used to fill a containerwith objects, create a stack of objects, or unload objects from a truckbed. In some cases, all of the objects may be of the same type. In othercases, a container or truck may contain a mix of different types ofobjects, such as boxed items, cans, tires, or other stackable objects.Such robotic systems may direct a robotic manipulator to pick up objectsbased on predetermined knowledge of where objects are in theenvironment.

In some examples, a robotic system may use computer vision techniques todetermine a representation of three-dimensional (3D) scene geometry. Byway of example, a robotic system may triangulate information observedfrom a scene to determine a depth to one or more surfaces in a scene.One approach to depth sensing is the use of stereo image processing.According to this approach, two optical sensors with a known physicalrelationship to one another are used to capture two images of a scene.By finding mappings of corresponding pixel values within the two imagesand calculating how far apart these common areas reside in pixel space,a computing device can determine a depth map or image usingtriangulation. The depth map or depth image may contain informationrelating to the distances of surfaces of objects in the scene.

Another approach to depth sensing using structured-light processing maybe employed. The main idea of structured-light processing is to projecta known illumination pattern onto a scene, and capture an image of thescene that includes the projected pattern. For example, as shown in FIG.1, a projector 102 may project a known texture pattern onto an object104, and an optical sensor 106 (e.g., a camera) may capture an image 108of the object 104. A computing device may then determine acorrespondence between a region in the image and a particular part ofthe projected pattern. Given a position of the projector 102, a positionof the optical sensor 106, and the location of the region correspondingto the particular part of the pattern within the image 108, thecomputing device may then use triangulation to estimate a depth to asurface of the object 104.

Typically the projector 102 and optical sensor 106 are displacedhorizontally along a baseline, and the projector 102 and optical sensor106 are calibrated. The calibration process may map a pixel in theoptical sensor 106 to a one-dimensional curve of pixels in the projector102. If the sensor image and the projector image are rectified, thenthis curve may take the form of a horizontal line. In this case, thesearch for matches to the projected texture pattern can proceed alongthis line, making the process more efficient.

SUMMARY

Depth sensing in environments that environments with bright and dimportions can be improved using an imager capable of sensing bothprojected patterns and visible light. As described herein, a pair ofimagers may capture one or more infrared light images of the projectedpattern and a pair of visible light images of the environment. Acomputing device may determine a first depth estimate for a surface in ascene from at least one of the infrared light images usingstructured-light processing. The computing device may also determine asecond depth estimate for a surface in the scene from the pair of stereoimages using stereo image processing. The computing device may thencombine the first and second depth estimates to determine a combineddepth estimate for one or more surfaces in the environment.

In one example, a system that includes a first optical sensor, a secondoptical sensor, a light source, and a computing device is provided. Eachoptical sensor includes a first plurality of photodetectors configuredto capture visible light interspersed with a second plurality ofphotodetectors configured to capture infrared light within a particularinfrared band. The light source is configured to project infrared lightof a wavelength within the particular infrared band onto an environment.The computing device is configured to identify first correspondingfeatures of the environment between a first visible light image capturedby the first optical sensor and a second visible light image captured bythe second optical sensor. The computing device is also configured toidentify second corresponding features of the environment between afirst infrared light image captured by the first optical sensor and asecond infrared light image captured by the second optical sensor. Thecomputing device is further configured to determine a depth estimate forat least one surface in the environment based on the first correspondingfeatures and the second corresponding features.

In another example, an optical sensor that includes a planar array ofphotodetectors, a light filter array, and a control circuit is provided.The planar array of photodetectors includes a first plurality ofphotodetectors and a second plurality of photodetectors. Eachphotodetector is configured to generate a charge based on an intensityof light incident on the photodetector. The light filter array iscoupled to the planar array of photodetectors. The light filter arrayincludes a first plurality of filters configured to pass visible lightonto the first plurality of photodetectors and a second plurality offilters configured to pass infrared light within a particular infraredband onto the second plurality of photodetectors. The first plurality offilters is interspersed with the second plurality of filters. Thecontrol circuit is configured to generate a visible light image based oncharges from the first plurality of photodetectors. The control circuitis also configured to generate an infrared light image based on chargesfrom the second plurality of photodetectors.

In still another example, a method is provided that involves receiving,from a first plurality of photodetectors of a first optical sensor, afirst visible light image of an environment as perceived from a firstviewpoint. The method also involves receiving, from a first plurality ofphotodetectors of a second optical sensor, a second visible light imageof the environment as perceived from a second viewpoint. The methodfurther involves receiving, from a second plurality of photodetectors ofthe first optical sensor, a first infrared light image of theenvironment as perceived from the first viewpoint. The second pluralityof photodetectors of the first optical sensor is configured to captureinfrared light within a particular infrared band. Additionally, themethod involves receiving, from a second plurality of photodetectors ofthe second optical sensor, a second infrared light image of theenvironment as perceived from the second viewpoint. The second pluralityof photodetectors of the second optical sensor is configured to captureinfrared light within the particular infrared band. Further, the methodinvolves identifying, by the computing device, first correspondingfeatures of the environment between the first visible light image andthe second visible light image. The method also involves identifying, bythe computing device, second corresponding features of the environmentbetween the first infrared light image and the second infrared lightimage. The method further involves determining, by the computing device,a depth estimate for at least one surface in the environment based onthe first corresponding features and the second corresponding features.

In still another example, a system is provided that includes means forreceiving, from a first plurality of photodetectors of a first opticalsensor, a first visible light image of an environment as perceived froma first viewpoint. The system also includes means for receiving, from afirst plurality of photodetectors of a second optical sensor, a secondvisible light image of the environment as perceived from a secondviewpoint. The system further includes means for receiving, from asecond plurality of photodetectors of the first optical sensor, a firstinfrared light image of the environment as perceived from the firstviewpoint. The second plurality of photodetectors of the first opticalsensor is configured to capture infrared light within a particularinfrared band. Additionally, the includes means for receiving, from asecond plurality of photodetectors of the second optical sensor, asecond infrared light image of the environment as perceived from thesecond viewpoint. The second plurality of photodetectors of the secondoptical sensor is configured to capture infrared light within theparticular infrared band. Further, the system includes means foridentifying, by the computing device, first corresponding features ofthe environment between the first visible light image and the secondvisible light image. The system also includes means for identifying, bythe computing device, second corresponding features of the environmentbetween the first infrared light image and the second infrared lightimage. The system further includes means for determining, by thecomputing device, a depth estimate for at least one surface in theenvironment based on the first corresponding features and the secondcorresponding features.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a conceptual illustration of an example approach forstructured-light processing.

FIG. 2A shows a robotic arm mounted on a moveable cart, according to anexample embodiment.

FIG. 2B is a functional block diagram illustrating a robotic device,according to an example embodiment.

FIG. 3A shows a robotic arm and a stack of boxes, according to anexample embodiment.

FIG. 3B shows scanning of the stack of boxes from FIG. 3A by a sensormounted on the robotic arm, according to an example embodiment.

FIG. 3C shows the robotic arm from FIG. 3A moving a box, according to anexample embodiment.

FIG. 4 is a conceptual illustration of the functionality of an exampleoptical sensor, according to an example embodiment.

FIG. 5A is a conceptual illustration of an example arrangement ofphotodetectors on an example optical sensor, according to an exampleembodiment.

FIG. 5B is a conceptual illustration of another example arrangement ofphotodetectors on an example optical sensor, according to an exampleembodiment.

FIG. 6 is a conceptual illustration of an example stereo imaging system,according to an example embodiment.

FIG. 7 is a block diagram of example functions that may be performed bya computing device, according to an example embodiment.

FIG. 8A is a conceptual illustration of an example configuration fordetermining a correlation surface, according to an example embodiment.

FIG. 8B is an example illustration of a correlation surface, accordingto an example embodiment.

FIG. 8C is an example illustration of combining correlation surfaces,according to an example embodiment.

FIG. 9 is a conceptual illustration of combining information frommultiple depth images, according to an example embodiment.

FIG. 10A is a conceptual illustration of an example environment,according to an example embodiment, according to an example embodiment.

FIG. 10B is an example illustration of determining depth from multipledepth images, according to an example embodiment.

FIG. 11A is an example image of a scene, according to an exampleembodiment.

FIG. 11B is an example depth image of the scene shown in FIG. 11A,according to an example embodiment.

FIG. 11C is an example image of a scene with a projected pattern,according to an example embodiment.

FIG. 11D is an example depth image of the scene shown in FIG. 11C,according to an example embodiment.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein. Furthermore, theparticular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments might includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anexample embodiment may include elements that are not illustrated in theFigures.

Methods, optical sensors, and robotic systems that may improve depthsensing in environments with a wide luminance range are provided herein.A robotic system equipped with a computer vision system may operatewithin a variety of environments with varying levels of lighting.Certain environments—such as outdoor environments during the daytime andwell-lit indoor environments—may be well suited for depth sensing usingstereo image processing. When objects in environments are brightlyilluminated, an imager sensing light reflected off those objects cancapture considerable detail of those objects' features (e.g. shape,color, contours, edges, etc.). However, in other environments—such asindoor environments with dim artificial lighting and nighttimeenvironments—less light may be reflected off objects, making discernmentof those objects' features more difficult. Furthermore, because stereoimage-based depth sensing primarily depends upon identifying commonfeatures between two images taken from different perspectives, it may bedifficult to determine the depth of objects that have few detectablefeatures, such as those without many color variations, edges, contours,or other artifacts (e.g. large, single-colored flat surfaces).

In dimmer environments and/or environments with objects having fewdetectable features, a structured-light approach can improve depthsensing by projecting patterned light onto a scene and capturing imagesof that scene with the projected pattern. The projected pattern shinesartificial features onto objects and surfaces that might otherwise havetoo few features or be too dimly lit for accurate depth sensing. Thus,by employing structured-light processing to images capturing anenvironment with a projected pattern, accurate depth sensing may beachieved for surfaces in an environment that might otherwise be too darkor have too few features. In some implementations, the projected patternmay be light of a wavelength within a particular infrared wavelengthband. While this structured-light technique may be useful in darker andsome indoor environments, the projected pattern may be washed out orovercome by a different light source, such as sunlight.

During operation, a robotic system may transition between bright anddark environments and/or from indoor to outdoor environments. In somecases, a robotic system may encounter environments that have acombination of bright portions and dark portions. For instance, apartially indoor and partially outdoor scene might have bright sunlitsurfaces contrasted against dark shadows. In these scenarios, a roboticsystem might benefit from depth sensing using a combination of stereoimage processing and structured light processing.

An optical sensor of the present application may be capable of detectinglight in one or more wavelength bands (e.g. a visible light band and/oran infrared light band). According to various embodiments, the opticalsensor includes a combination of visible-sensing photodetectorsinterspersed with infrared-sensing photodetectors. The visible-sensingand infrared-sensing photodetectors can be arranged in a variety ofmanners discussed herein. Control circuitry of the optical sensor maygenerate visible light images using the visible-sensing photodetectorsand infrared light images using the infrared-sensing photodetectors. Asa result, one such optical sensor may capture images in two differentwavelength bands of light.

A single optical sensor capable of capturing both visible light imagesand infrared light images may be more resource efficient compared to twoseparate optical sensors that each independently capture either visiblelight images or infrared light images. Reducing the number of opticalsensors may be desired when space resources are limited, such as onsmall robotic systems. Furthermore, the single optical sensor mayrequire less energy to operate and might increase the reliability of therobotic system by reducing the number of sensors susceptible to failure.

In addition to the above-described benefits, a single optical sensorthat intermixes both visible-sensing photodetectors and infrared-sensingphotodetectors reduces the effects of the systematic errors that wouldotherwise be present if two separate optical sensors were used.Furthermore, certain interspersed arrangements of the visible-sensingphotodetectors and the infrared-sensing photodetectors may be employedto avoid or further reduce the effects of the systematic errors. Thus,using a single optical sensor avoids the need to perform error-pronecalibration techniques typically associated with using two differentoptical sensors.

According to various embodiments, an imaging system includes two or moreof the above-described optical sensors (which may also be referred toherein as “imagers”) configured to capture images of an environment fromdifferent viewpoints. The imaging system might stereoscopically image anenvironment or scene in multiple wavelength bands. Then, a computingdevice may use stereo image processing to generate multiple depth maps(e.g. depth information of surfaces) of the environment based on pairsof images from different viewpoints in each wavelength band. Bycombining the depth information derived from multiple wavelength bandsinto a single depth map, the computing device may generate a morecomplete and accurate depth map of an environment with a wide luminancerange.

According to various embodiments, an imaging system may include two ofthe above-described optical sensors separated from each other by a knowndistance and each configured to capture images in both the visible lightband and a particular infrared (IR) light band. The imaging system mayalso include a texture projector that projects a known structured-lightpattern onto the environment within that particular IR light band. Whilethis IR pattern projects features onto surfaces that improve depthdetermination (especially in low-light environments, where features ofobjects may be difficult to discern in the visible light spectrum), theprojected pattern may be overcome or washed out by intense infraredlight from other sources, such as artificial lighting and/or sunlight.As a result, some features—such as color differences, edges of objects,and/or portions of the projected pattern—may only be perceived in eitherthe visible light band or the particular IR light band.

In one example, a computing device may determine depth information of anenvironment based on corresponding features between a pair of visiblelight images captured by the imaging system. By way of example, thecomputing device may determine mappings of corresponding pixel valueswithin the two visible light images, and based on a physicalrelationship between the two optical sensors, the computing device candetermine depth information using triangulation. In environments with awide luminance range, the visible light-based depth information mayprovide depths for the well-illuminated portions of the environment. Thecomputing device may also determine depth information of the environmentbased on corresponding features between a pair of IR light imagescaptured by the imaging system. The IR light-based depth information mayprovide depths for the dimly-lit portions of the environment. Then, thecomputing device may combine the depth information derived from both thevisible light images and the IR light images to determine a morecomprehensive and accurate depth map of the surfaces in the environment.

Various additional examples are described below, therefore the aboveexamples are not meant to be limiting. Reference will now be made indetail to various embodiments, examples of which are illustrated in theaccompanying drawings. In the following detailed description, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present disclosure and the described embodiments.However, the present disclosure may be practiced without these specificdetails. In other instances, well-known methods, procedures, components,and circuits have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

Additionally, although portions of this disclosure refer to implementingthe described methods and apparatuses to facilitate manipulation (e.g.,loading and/or unloading) of boxes and/or other objects, the methods andapparatuses may be implemented in other environments as well. Forinstance, the methods and apparatuses may be implemented generally inany environment to determine the 3D geometry of the environment or 3Dgeometry of objects in the environment. By way of example, the methodsand apparatuses may be implemented to help determine the 3D geometry ofa room in a house or an office in building, or perhaps the 3D geometryof a portion of an arena, conference center, or shopping mall.Similarly, the methods and apparatuses may be utilized in outdoorenvironments as well.

According to various embodiments, described herein are methods andsystems for automated loading and/or unloading of boxes and/or otherobjects, such as into a storage container or from a vehicle. In someexample embodiments, boxes or objects may be automatically organized andplaced onto pallets (palletized) or automatically removed from pallets(depalletized). Within examples, automating the process ofloading/unloading trucks and/or the process of palletizing/depalletizingobjects may provide a number of industrial and business advantages.

According to various embodiments, automating the process ofloading/unloading trucks and/or the process ofpalletizing/un-palletizing objects may include the incorporation of oneor more robotic devices to move objects or perform other functions. Insome embodiments, a robotic device can be made mobile by coupling therobotic device with a wheeled base, a holonomic base (e.g., a base thatcan move in any direction), or rails on the ceiling, walls or floors. Insome embodiments, the base can be an elevated base.

In some examples, a system including one or more sensors, one or morecomputers, and one or more robotic arms is described. The sensors mayscan an environment containing one or more objects in order to capturevisual data and/or 3D depth information. Data from the scans may then beintegrated into a representation of larger areas in order to providedigital environment reconstruction. In additional examples, thereconstructed environment may then be used for identifying objects topick up, determining pick positions for objects, and/or planningcollision-free trajectories for the one or more robotic arms and/or amobile base.

As used herein, the term “boxes” will refer to any object or item thatcan be placed onto a pallet or loaded onto or unloaded from a truck orcontainer. For example, in addition to rectangular solids, “boxes” canrefer to cans, drums, tires or any other “simple” shaped geometricitems. Additionally, “loading” and “unloading” can each be used to implythe other. For example, if an example describes a method for loading atruck, it is to be understood that substantially the same method canalso be used for unloading the truck as well. As used herein,“palletizing” refers to loading boxes onto a pallet and stacking orarranging the boxes in a way such that the boxes on the pallet can bestored or transported on the pallet while the term “depalletizing”refers to removing boxes from a pallet. In addition, the terms“palletizing” and “depalletizing” can each be used to imply the other.

According to various embodiments, a robotic manipulator may be mountedon a holonomic cart (e.g., a cart with wheels that allow the cart tomove in any direction). FIG. 2A depicts an exemplary holonomic cartcontaining a robotic manipulator. In some embodiments, a moveable cart212 may include a robotic arm 202 mounted on the cart 212. The roboticarm 202 may contain a gripping component 204 for gripping objects withinthe environment. The cart may contain one or more wheels 214, which maybe holonomic wheels that operate with two degrees of freedom. In furtherembodiments, a wraparound front conveyor belt 210 may be included on theholonomic cart 212. In some examples, the wraparound front conveyer belt210 may allow the robot to not have to rotate its gripper to the left orright when unloading or loading boxes from or to a truck container orpallet.

In other examples, the robotic manipulator may be mounted on a differenttype of movable apparatus or may not be mounted on a movable base atall. For example, the robotic manipulator may be mounted at a fixedposition within a factory setting. In other example embodiments, one ormore robotic manipulators may be mounted on rails of a truck orcontainer. In such examples, the robotic manipulators may be used toload or unload the truck or container.

FIG. 2B is a functional block diagram illustrating a robotic device 200,according to an example embodiment. The robotic device 200 could includevarious subsystems such as a mechanical system 220, a sensing system230, a control system 240, as well as a power supply 250. The roboticdevice 200 may include more or fewer subsystems and each subsystem couldinclude multiple elements. Further, each of the subsystems and elementsof robotic device 200 could be interconnected. Thus, one or more of thedescribed functions of the robotic device 200 may be divided up intoadditional functional or physical components, or combined into fewerfunctional or physical components. In some examples, additionalfunctional and/or physical components may be added to the examplesillustrated by FIGS. 2A and 2B.

The mechanical system 220 may include components described above withrespect to FIG. 2A, including a robotic arm 202, a gripper 204, aconveyer belt 210, a (movable or holonomic) cart 212, and one or morewheels 214. The mechanical system 220 may additionally include a motor222, which may be an electric motor powered by electrical power, or maybe powered by a number of different energy sources, such as a gas-basedfuel or solar power. Additionally, motor 222 may be configured toreceive power from power supply 250. The power supply 250 may providepower to various components of robotic device 200 and could represent,for example, a rechargeable lithium-ion or lead-acid battery. In anexample embodiment, one or more banks of such batteries could beconfigured to provide electrical power. Other power supply materials andtypes are also possible.

The sensing system 230 may use one or more sensors attached to a roboticarm 202, such as sensor 206 and sensor 208, which may be 2D sensorsand/or 3D depth sensors that sense information about the environment asthe robotic arm 202 moves. The sensing system may determine informationabout the environment that can be used by control system 240 (e.g., acomputer running motion planning software) to pick and move boxesefficiently. The control system 240 could be located on the device orcould be in remote communication with the device. In further examples,scans from one or more 2D or 3D sensors mounted on a mobile base, suchas a front navigation sensor 216 and a rear navigation sensor 218, andone or more sensors mounted on a robotic arm, such as sensor 206 andsensor 208, may be integrated to build up a digital model of theenvironment, including the sides, floor, ceiling, and/or front wall of atruck or other container. Using this information, the control system 240may cause the mobile base to navigate into a position for unloading orloading.

In additional examples, planar surface information may be extracted from3D sensors to model walls, the floor/ceiling, and/or box faces. Aftermodeling the floor, projecting objects onto the floor plane may enablesegmentation of obstacles and/or target objects such as boxes.Floor-plane projection can also be used to model the corrugated sides ofa container or truck, which may not be accurately modeled as planes. Infurther examples, sidewall angles, floor plane roll and pitch, and/ordistances from side walls can be used to maneuver a mobile base into acontainer without collisions. Use of extended 3D information rather thana single line scan may help make the extraction of navigationinformation robust. For example, the side walls may have a verticalextent that is captured by the 3D sensor. Scanning systems that use asingle line of depth information may be slower if they scan verticallyand/or less robust because they acquire less information. In additionalexamples, front plane modeling can determine the distance to a nextgroup of objects to pick in truck unloading.

In further examples, the robotic arm 202 may be equipped with a gripper204, such as a digital suction grid gripper. In such embodiments, thegripper may include one or more suction valves that can be turned on oroff either by remote sensing, or single point distance measurementand/or by detecting whether suction is achieved. In additional examples,the digital suction grid gripper may include an articulated extension.In some embodiments, the potential to actuate suction grippers withrheological fluids or powders may enable extra gripping on objects withhigh curvatures.

In some embodiments, the gripper could potentially span several boxes orobjects and turn on suction for some or all of the covered objects. Insome embodiments, the suction or adhesion devices may be a “digital”grid so that the robotic device can turn on any number of the suctiondevices as will fit boxes sensed for grabbing. In some implementations,the system may notice a seam in the boxes (separation between adjacentboxes) such that suction devices can be activated on both sides of theseam to pick up both boxes at once, thereby doubling the throughput. Insome embodiments, the suction devices can sense after a certain amounttime whether they can successfully grip a surface, after which they mayautomatically shut off. In further embodiments, sections of the suctiondevices can fold down to grasp the top of the boxes. For instance,grippers can initially start at full extension and then conform to thesurface being gripped.

In further examples, the robotic arm can implement a wiggle movement toimprove a suction grip. In additional embodiments, the robotic arm canwiggle a box side-to-side to help segment the box from its surroundings.In other embodiments, the arm can wiggle upon pick up of the box toavoid jostling other objects. In such embodiments, when trying to adhereto an object in order to pick it up using suction, the robotic arm mayemploy a wiggle motion in order to make a firm seal against the object.In further examples, the robotic arm may wiggle the object as therobotic arm is picking up the object so that the box can more gentlybreak friction or overlap contact with other items. This may help avoida situation where pulling the object up too directly or too quicklycauses other items to be tossed into the air.

According to various embodiments, cardboard boxes can have concave,convex or otherwise rumpled faces that make it hard for a suction deviceto adhere to. Thus, wiggling the suction device as the device makessuction contact may enable a more reliable grip on cardboard boxes andother non-planar objects. In further examples, when first grabbing abox, a few center suction devices can be turned on and the arm canwiggle back and forth as it starts to pull the box out. This may breaksurface adhesion with other boxes and help to start to pull the box out.Once the box is at least partially pulled out, the box may then besegmented from the other boxes more easily. In some embodiments,wiggling while picking up an object in clutter may remove other objectsfrom the picked up object, thereby preventing unwanted pick up ofsurrounding objects.

According to various embodiments, segmentation of items may be necessaryfor successful grasping. In some embodiments, a smooth surface patch maybelong to two separate objects. In such instances, manipulatorinteraction with the objects may be used to perturb the scene to bettersegment the objects from each other. For motion separation, the naturalor forced movement of objects on a conveyor, on a slide, moving in atote, and/or actively jostled within a tote may be tracked by opticalflow, parallax, or time delayed views to calculate stereo depth in orderto enhance object segmentation.

In other examples, one or more of the sensors used by a sensing systemmay be a RGBaD (RGB+active Depth) color or monochrome camera registeredto a depth sensing device that uses active vision techniques such asprojecting a pattern into a scene to enable depth triangulation betweenthe camera or cameras and the known offset pattern projector. This typeof sensor data may help enable robust segmentation. According to variousembodiments, cues such as barcodes, texture coherence, color, 3D surfaceproperties, or printed text on the surface may also be used to identifyan object and/or find its pose in order to know where and/or how toplace the object (e.g., fitting the object into a fixture receptacle).In some embodiments, shadow or texture differences may be employed tosegment objects as well.

Many or all of the functions of robotic device 200 could be controlledby control system 240. Control system 240 may include at least oneprocessor 242 (which could include at least one microprocessor) thatexecutes instructions 244 stored in a non-transitory computer readablemedium, such as the memory 246. The control system 240 may alsorepresent a plurality of computing devices that may serve to controlindividual components or subsystems of the robotic device 200 in adistributed fashion.

In some embodiments, memory 246 may contain instructions 244 (e.g.,program logic) executable by the processor 242 to execute variousfunctions of robotic device 200, including those described above inconnection with FIGS. 2A-2B. Memory 246 may contain additionalinstructions as well, including instructions to transmit data to,receive data from, interact with, and/or control one or more of themechanical system 220, the sensor system 230, and/or the control system240.

According to various embodiments, a perception guided robot isdescribed. For instance, a robotic device may utilize a combination ofperception together with planning to guide the robot arm to pick up abox and place the box where it needs to go. FIG. 3A illustrates part ofthe robotic device from FIG. 2A with a stack of boxes, according to anexample embodiment. As shown, the robotic device may include a roboticarm 202 with a gripping component 204, sensors 206 and 208, and conveyer210 as described above. In some examples, the robotic device could bemounted on a holonomic cart as described with respect to FIG. 2A, couldbe mounted on a different type of movable apparatus, could be mounted onrails or tracks, or could be stationary. The robotic device may becontrolled to pick boxes from a stack of boxes 320 containing aheterogeneous mix of shapes and sizes of boxes.

Within examples, a virtual environment including a model of the objectsin 2D and/or 3D may be determined and used to develop a plan or strategyfor picking up the boxes. In some examples, the robot may use one ormore sensors to scan an environment containing objects, as shown in FIG.3B. As the robotic arm 202 moves, a sensor 206 on the arm may capturesensor data about the stack of boxes 320 in order to determine shapesand/or positions of individual boxes. In additional examples, a largerpicture of a 3D environment may be built up by integrating informationfrom individual (e.g., 3D) scans. Sensors performing these scans may beplaced in fixed positions, on a robotic arm, and/or in other locations.According to various embodiments, scans may be constructed and used inaccordance with any or all of a number of different techniques.

In some examples, scans can be made by moving a robotic arm upon whichone or more 3D sensors are mounted. Feedback from the arm position mayprovide pose information about where the sensor is positioned and may beused to help with the integration. Alternately, or additionally, scansmay be made using one or more 2D sensors, for instance by leveragingmotion and tracking key-points in the environment. In further examples,scans may be from fixed-mount cameras that have fields of view (FOVs)covering a given field. In additional examples, scans may be visuallyregistered to help with fine pose estimation, potentially giving betterintegration results.

In further examples, a virtual environment may be built up using a 3Dvolumetric or surface model to integrate information (e.g., fromdifferent sensors). This may allow the system to operate within a largerenvironment, such as in cases where one sensor may be insufficient tocover a large environment. Such techniques may also increase the levelof detail captured, which may help the robotic device perform varioustasks. In particular, integrating information can yield finer detailthan from a single scan alone (e.g., by bringing down noise levels).This may enable better object detection, surface picking, or otherapplications.

In further examples, wide-angle environment reconstruction may beperformed by sensing an environment and extracting that information intoa simplified geometric model of simple mathematical 3D geometric forms(e.g., planes, cylinders, cones, hemispheres, etc.). In some instances,such techniques may make motion planning easier and/or may makeviolation of the models (e.g., collisions) easier to detect.Alternately, or additionally, such techniques may allow a parametricdescription to extend the environment. For instance, the ground may betreated as a plane that extends behind objects that occlude it.

In additional examples, planes or other mathematical surfaces in theenvironment may be extracted in 3D. These known “ideal” surfacedetections may be combined into a more accurate model of theenvironment. For instance, planes may be used to determine the fullextents of walls (or mathematical description thereof) and otherobstacles to avoid collisions and detect the locations of objects ofinterest. Also, mathematical representations of objects may be used tolook for anomalies such as a person entering into an environment. Suchevents may violate the ideal model, which may make their detectioneasier.

In other examples, certain objects such as boxes may have simple planarform. For instance, a metal can may have a geometric form of a cylinderand a tire may have a geometric form of a torus. Example systems mayleverage this trait of certain objects in order to model them and/ordetermine how to motion plan for the objects. For instance, knowntemplates of certain shapes can be used to refine detected features ofobjects within the environment that appear to match a particular shape.

In some examples, 2D and 3D information may be represented at least inpart via one or more facades. A facade may be defined as a near-planarconstruct containing a set of objects, represented as a depth map (e.g.,a 2D map with distances as a third dimension). Examples of facades mayinclude a wall of boxes in a truck, a top of a pallet stack containingboxes or other objects, or a top of a bin of jumbled objects.

In further examples, a facade may be constructed from boxes, forinstance to plan an order for picking up the boxes. For instance, asshown in FIG. 3C, box 322 may be identified by the robotic device as thenext box to pick up. Box 322 may be identified within a facaderepresenting a front wall of the stack of boxes 320 constructed based onsensor data collected by one or more sensors, such as sensors 206 and208. A control system may then determine that box 322 is the next box topick, possibly based on its shape and size, its position on top of thestack of boxes 320, and/or based on characteristics of a targetcontainer or location for the boxes. The robotic arm 202 may then becontrolled to pick up the box 322 using gripper 204 and place the box322 onto the conveyer belt 210 (e.g., to transport box 322 into astorage area).

In additional examples, a facade may be represented as an orthographicprojection of 3D surface information. This representation may allow forparsing the facade to determine interesting areas for a particularapplication. For example, in truck unloading, the upper left corner ofthe next box to pick up may be determined based on a facaderepresentation. In other examples, an orthographic projection of anintegrated 3D environment may be determined to give a wide-FOV,easily-parsed representation for performing application-related tasks.One such task may be finding the corner or corners (e.g., top left) of abox to pick. Another such task may involve finding good surfaces (e.g.,relatively flat and large) for picking objects out of a bin.

In further examples, a 3D model of a stack of boxes may be constructedand used as a model to help plan and track progress forloading/unloading boxes to/from a stack or pallet. Any one actual cameraview of the facade may suffer from point of view occlusions andperspective distortion. Accordingly, multiple RGBD views via robot armmovements and/or different views from a cart base or fixed locations maybe combined to create a single facade of the boxes to be picked.

In other examples, the 3D model may be used for collision avoidance.Within examples, planning a collision-free trajectory may involvedetermining the 3D location of objects and surfaces in the environment.A trajectory optimizer may make use of the 3D information provided byenvironment reconstruction to optimize paths in the presence ofobstacles. In further examples, the optimizer may work in real time andmay accept many kinds of constraints. As an example of such aconstraint, the optimizer may attempt to keep the end effector levelthroughout the trajectory.

In additional examples, an environment may be captured as a mesh or setof 3D points. A robot arm may be represented as a convex hull of planesegments for quick collision checking. Constant or frequent updating ofthe environment may allow the robot arm to quickly respond to changes.In further examples, an optimizer may perform frequent continuouscollision checking throughout its path. An optimizer may acceptarbitrary constraints in the form of costs, such as to keep a certaindistance away from objects or to approach a goal position from a givenangle. Additionally, an optimizer may avoid robot fault conditions byworking in joint space, keeping track of windup and choosing goalpositions from among multiple inverse kinematics solutions. One strategyfor motion planning may involve looking ahead several moves to see ifthe chosen goal joint position will be acceptable for the next move.

In some embodiments, path constraints, such as collision avoidance forrobotic arms, cameras, cables, and/or other components, may be put in aconstraint based planning solver and solved for to yield a best path tomove the arm for perception. Additionally, in some embodiments, thesolver may determine a best path for picking up, moving, and placing anobject.

According to various embodiments, 3D and/or visual sensors may becalibrated to determine their pose with respect to the workspace. In thecase of fixed sensors, the calibration may determine their fixed pose inthe workspace. In the case of a sensor on the arm, calibration maydetermine the offset pose of the sensor from the arm link to which it isattached.

Within examples, calibration techniques may allow for the calibration ofan arbitrary number of sensors in the workspace. Calibration may involvedetermining some or all of a variety of parameters and coefficients. Forexample, calibration may solve for one or more intrinsic parameters suchas focal length and image center. As another example, calibration maydetermine one or more distortion coefficients such as models of radialand tangential distortion. As yet another example, calibration may solvefor one or more extrinsic parameters defining a position of an object ina scene relative to a pattern or other sensors that identified the samepattern in a scene.

In some examples, calibration may be performed at least in part by usinga calibration pattern, which may be a known set of features in 2D or 3D.For instance, a known pattern of dots may be used, where the distancebetween each dot and the other dots is known. Calibration may beperformed at least in part by collecting multiple different views of anobject. In further examples, capturing multiple views of a calibrationpattern in different positions may allow for (1) calibration of the oneor more coefficients of the camera and/or (2) knowledge of where thecamera is relative to the coordinate system established by where thecalibration pattern was fixed. In particular embodiments, a camera inthe scene may identify a calibration pattern on the robot arm while acamera on the arm identifies a calibration pattern in the scenesimultaneously.

In additional examples, calibration may involve a camera fixed in ascene. In this case, a calibration pattern may be placed on a roboticarm. The robotic arm may be configured to move through the scene asmultiple views of the calibration pattern on the robotic arm arecollected. This may help to calibrate the camera and/or be useful forrelating the coordinate system of the camera to that of the robot.Further, the relation of each device to the other can be determined byeach device as the robotic arm moves.

In certain examples, calibration may involve a camera located on arobotic arm. A calibration pattern may be mounted on a wall or table.Then, the camera may be moved around, collecting multiple views of thecalibration pattern from different robot or robotic arm positions. Whendifferent 3D or 2D views (e.g., 2, 20, or 200) are collected, theseviews can be used to solve for the calibration relationships. Aftercalibration, when the camera on the arm moves, the system can determinewhere it is relative to the coordinate system set based on the locationof the calibration pattern in the scene. In particular embodiments, boththe calibration pattern and the camera may be movable. For example, thecalibration pattern may be located on a conveyor belt where the roboticarm may be configured to place boxes. After calibration, the system maydetermine where the camera was relative to that spot on the conveyorbelt.

In further examples, nonlinear optimization may be performed in atwo-stage process for robust estimation of 3D sensor calibration. In onestage, an initialization may be derived from the relative pose offsetsof the target and the sensors. In another stage, given theinitialization, a batch bundle adjustment may be used to find theoptimal pose of the cameras together with the target points. Calibrationcan be extended to the estimation of robot parameters such as jointlengths and joint angle offsets.

In other examples, known, precise, robot motion of the camera over acalibration pattern, or a calibration pattern over a camera may be usedto improve calibration results. For instance, information aboutprecisely how the camera moves may be used to obtain more accuratecamera calibration. That is, if the camera is moved 50 mm to the right,the corresponding (perspective projection) amount of movement from thecalibration object may be detected. This information may be used tojointly or separately optimize the calibration and tracking parameters.

In additional examples, a robot can look at its ongoing calibration andmove in such a way as to maximize information for better calibration.For example, it can detect that some view areas have not been seen andgo to those views.

In further examples, a system for the practical manipulation ofheterogeneous, categorical items, generally from a cluttered collectionarea to a defined bin, is presented. In some embodiments, the picklocation containing the items may not be sensitive to precise objectorientation(s) and items may be mixed together. In additional examples,the place location for the items may or may not be sensitive to objectorientation. In some examples, the pick-and-place regions may be definedas 3D regions acceptable for picking or placing an object, with sometolerance. The pick-and-place region may be highly cluttered withsimilar and/or disparate objects. In other embodiments, the items maycome from or be put into a fixture, such as metal or plastic snaps thathold the sorted item in a particular orientation.

In additional examples, environment modeling of both the pick-and-placelocation may be used for intelligent grasp location and motion, as wellas event reporting (e.g., when a place region is full or a pick regionis empty). In some examples, object bounding volumes may be computedand/or distinguishing features of objects may be found (such astextures, colors, barcodes or OCR). In some embodiments, objects may besorted into an assigned destination location by matching against adatabase of location assignments indexed by object type or object ID.For instance, an object's locations may be derived from reading abarcode, considering the size of the object, and/or by recognizing aparticular kind of object.

In some examples, a plan for a robotic device may be determined in orderto achieve certain configurations of the objects within a targetlocation for the objects. For instance, the goals for loading/unloadingor palletizing/depalletizing may be to achieve: (1) a dense packing withminimal air gaps in between boxes, and/or (2) a stable packing that willnot easily collapse. In some embodiments, stability may require that, ingeneral, heavy objects are on the bottom, and light objects are on top.In other examples, pallets may be created in order to avoidnon-interlaced column stacks, column leans, or other characteristics ofa bad stack.

In further examples, the pallet or truck/container may be loaded suchthat work by human operators in subsequent unloading processes isminimized. For instance, in some embodiments, items may be placed inlast in, first out order such that, upon unpacking, the items neededfirst are on top, the items needed second are one layer down and so on.In other examples, the loading of pallets may be independent of howitems flow towards the packing cells. Thus, according to someembodiments, the system can handle packages sent in random order or inan order known in advance. In addition, in some embodiments, systems mayadapt to changes in the flow of items in real-time. In further examples,one or more boxes may be recorded and buffered by holding the boxes in atemporary storage area where their order can be changed along the way.

According to various embodiments, a 2D simulator and/or a 3D simulatormay be utilized for truck or container loading/unloading or for palletloading/unloading. In some examples, the state of a stack of boxes maybe captured in the physical world and input into the simulator. In someembodiments, a variable size queue of boxes from one to all the boxesmay be used by a simulator for finding a next box to pick. For example,a queue of 2 boxes or 4 boxes or 10 boxes may be considered by asimulator.

In further examples, the simulator may search over the boxes in thequeue to find the best box placement by heuristic algorithms and/or bybrute force or multi-resolution search. In some embodiments, the systemmay increment with increasingly fine placement of boxes around the bestsites found in the previously coarser level. In some embodiments, onceplacement of a particular next box has been determined, a physicsplanner may be used for motion planning in order to move the boxefficiently to the determined locations. In further examples, thephysical and simulated stacks may be continuously monitored for qualityof the stack (e.g., density, stability, and/or order placement). In someexamples, the process may be repeated until all the boxes have beenplaced or the target container can no longer fit in another box.

FIG. 4 is a conceptual illustration 400 of the functionality of anexample optical sensor 402. The optical sensor 402 includes an array ofphotodetectors, such as photodetector 404. Some of the photodetectors inthe array are configured to generate a charge when exposed to anincident beam of light whose wavelength is within the visible spectrum(e.g. 380 nm to 750 nm). The other photodetectors in the array areconfigured to generate a charge when exposed to an incident beam oflight whose wavelength is within the infrared spectrum (e.g. 750 nm to3000 nm). In some embodiments, the infrared-sensing photodetectors areconfigured to capture a narrow band of infrared light (e.g. 825 nm to875 nm). In structured-light embodiments, an infrared texture projectormay project patterned light within such a narrow band of infrared light.

The visible-sensing photodetectors and the infrared-sensingphotodetectors may be arranged in a variety of manners. In someembodiments, the visible-sensing photodetectors may be interspersed withthe infrared-sensing photodetectors, such that there is a substantiallyeven mixture of each type of photodetector across the array ofphotodetectors. Various manners of arranging visible-sensing andinfrared-sensing photodetectors are contemplated herein and discussedfurther in more detail below.

In some implementations, the array of photodetectors may be coupled toan array of light filters. The light filters may be overlaid on top ofthe photodetectors such that light incident on a particularphotodetector first passes through a respective light filter. Each lightfilter may act as a band-pass filter that passes through light whosewavelength is within a particular band, while blocking or attenuatinglight whose wavelength is outside of that particular band. As describedherein, photodetectors that are “configured to” capture or detect acertain wavelength of light may be implemented as a photodetectorcoupled to such a light filter as described above.

During operation, visible light 406 and infrared (IR) light 408 incidenton optical sensor 402 may be captured by the optical sensor'svisible-sensing photodetectors and infrared-sensing photodetectors,respectively. The optical sensor may include control circuitry and/orprocessing devices configured to generate a visible light image 410 andan IR light image 412 from the charges in the photodetectors. Theseimages may then be provided to a computing device for use in stereoimage processing and depth sensing, among other possible uses.

Note that the particular wavelength bands with which the visible lightphotodetectors and the infrared light photodetectors capture light mayvary depending on the particular implementation. Additionally, the rateof drop off at the edges of these wavelength bands may also varydepending on the specific implementation. It should be understood thatthe boundaries of the visible-light band and the infrared-light bandmight vary among implementations.

Also note that “visible light image” as described herein may, in someinstances, refer to a panchromatic image (e.g. a grayscale imagesensitive to colors of light in the visible spectrum). In otherinstances, photodetectors may be provided for separately sensing red,green, and blue visible light. In such instances, an optical sensor'scontrol circuitry may perform demosaicing or other image processes inorder to construct a color image. “Visible light images” as describedherein may refer to either panchromatic images or color images.

FIG. 5A is a conceptual illustration of an example arrangement 500 ofphotodetectors on an example optical sensor, according to an exampleembodiment. The example arrangement 500 depicted in FIG. 5 may be oneexample arrangement of the photodetectors for optical sensors of thepresent application, such as the optical sensor 402 illustrated in FIG.4. The example arrangement 500 shown in FIG. 5 is a checkerboardpattern, where visible-sensing photodetectors and infrared-sensingphotodetectors alternate both row-wise and column-wise. However, itshould be understood that a variety of other photodetector patterns maybe used on a particular optical sensor, depending upon the particularimplementation.

During operation, the optical sensor may capture a visible light imageusing its visible-sensing photodetectors and an infrared light imageusing its infrared-sensing photodetectors. As described herein, theseimages may be used in stereo image processing to determine the depth ofa surface in an environment. Such stereo image processing may involvedetermining a pixel location of a feature within an image from oneperspective (e.g. from a first optical sensor) and comparing it to apixel location of that feature within an image from a differentperspective (e.g. from a second optical sensor). Using triangulation,the depth of that feature may be determined based on a known distancebetween the two different perspectives (e.g. distance between the firstand second optical sensor) and the relative pixel locations of thatfeature in the two images.

Depth sensing techniques of the present application involve performingsuch stereo image processing with two different types of light—infraredlight and visible light—to determine depth estimates in two differentbands (e.g. wavelength bands or frequency bands) of light. Then, thosedepth estimates may be combined to form a combined depth estimate for atleast one surface in an environment. However, because visible-sensingphotodetectors and the infrared-sensing photodetectors do not occupy theexact same positions on the photodetector but are rather interspersed ina planar array, some amount of inaccuracy may be present when combiningthe two separate depth estimates.

As one example, a visible light image may include a feature captured byvisible-sensing photodetector 504, while an infrared light image mayinclude that same feature captured by infrared-sensing photodetector506. This feature may appear to occupy the same pixel location in boththe visible light image and the infrared light image, although thephysical positions of the respective photodetectors that captured thefeature are different (in this example, the distance betweenphotodetector 504 and photodetector 506). This slight inaccuracy mayvary depending on the particular pattern with which the visible-sensingphotodetectors and infrared-sensing photodetectors are interspersed. Thecheckerboard pattern illustrated in FIG. 5 is one example arrangement ofphotodetectors that reduces such inaccuracy.

In addition to reducing such inaccuracy, the checkerboard patternillustrated in FIG. 5 provides reduced systematic error. Consider adifferent arrangement of photodetectors where visible-sensingphotodetector columns alternate with infrared-sensing photodetectorcolumns. An imaging system utilizing two such column-alternating opticalsensors might be subject to increased systematic error. If the twooptical sensors are poorly calibrated, the relative pixel positionswithin a captured image might be offset, resulting in a systematicerror. This offset could affect the accuracy of the pixel locations forentire columns of photodetectors. By alternating the photodetector typealong both rows and columns (e.g. the checkerboard pattern), such asystematic bias may be avoided or its effects significantly reduced.

In some embodiments, this inaccuracy may be accounted for or correctedusing software and/or techniques based on the known arrangement ofphotodetectors. Consider the photodetector arrangement 500 in FIG. 5 tobe the entire optical sensor, consisting of 16 photodetectors—8visible-sensing photodetectors and 8 infrared-sensing photodetectors. Inan example embodiment, the 8 visible-sensing photodetectors might beused to generate a 2×2 panchromatic image from light in the visiblespectrum, whereas the 8 infrared-sensing photodetectors might be used togenerate a 2×2 image from light in a narrow-band infrared spectrum. Inthis example, each mosaic (such as mosaic 502) might correspond to asingle pixel in an image; that is, the two visible-sensingphotodetectors correspond to a pixel in the visible light image, whilethe two infrared-sensing photodetectors correspond to a pixel in theinfrared light image. Charges generated by two photodetectors of thesame type within a mosaic might be averaged when generating the pixelfor an image. Although the resolution of the resulting image is reduced(because two photodetectors contribute to a single pixel), systematicbias is reduced; light from a feature captured by a particular mosaic isrepresented at the same pixel location in the visible light image andthe infrared light image.

In other embodiments, the visible-sensing photodetectors may be arrangedin an alternating pattern with the infrared-sensing photodetectors. Asone example, each row in the array may alternate between visible-sensingand infrared-sensing photodetectors. As another example, each column inthe array may alternate between visible-sensing and infrared-sensingphotodetectors. In additional embodiments, the optical sensor mayarrange the visible-sensing and infrared-sensing photodetectors in arepeating pattern or mosaic. For example, various square tiling patternsmay be implemented that combine any number of visible-sensing andinfrared-sensing photodetectors within a mosaic. A given mosaic mayinclude any number of photodetectors. Regardless of the particulararrangement of photodetectors, the optical sensor may have somecombination of visible-sensing and infrared-sensing photodetectors thatenable the optical sensor to capture both visible light images andinfrared light images.

In some implementations, a robotic system utilizing an visible light-and infrared light-sensing optical sensor may be aware of, detect, orotherwise classify the environment it is currently in at a given moment.Such a robotic system may alter the operation of one or morephotodetectors of the optical sensor based on the determined environmentin which it is currently presiding. The control circuit of the opticalsensor may be configurable during operation to change variousoperational aspects and parameters of the optical sensor, including anamount of gain to apply to the charges of each optical sensor and alength of time with which to expose the photodetectors to light (theexposure time), among other possible operational aspects. Different gainvalues and exposure times may be set for the visible-sensingphotodetectors and the infrared-sensing photodetectors, in someimplementations.

As one example, a robotic system may determine that it is operating in adaytime outdoor environment and responsively reduce the exposure timeand/or gain values of the photodetectors. In some cases, the roboticsystem may reduce or otherwise set the exposure time of thevisible-sensing photodetectors, while not reducing or otherwise changingthe exposure time of the infrared-sensing photodetectors. If thatrobotic system moves into a dark indoor environment, the robotic systemmay responsively increase the exposure time and/or gain values of thevisible-sensing photodetectors and/or the infrared-sensingphotodetectors. The robotic system may encounter other scenarios thatmight require varying the gain values and/or exposure time of thevisible-sensing photodetectors, the infrared-sensing photodetectors, orboth.

In some instances, a robotic system may be designed to operate within aparticular environment and is not expected to experience drastic changesin lighting. For example, a robotic system may be designed to operatesolely in an indoor environment with few windows with relativelyconsistent artificial lighting. However, such an environment might stillcontain well-lit surfaces and shadowed surfaces, although the expectedluminance range is smaller than in other mixed indoor and outdoorenvironments. A robotic system in such an environment might employ oneor more optical sensors that each contains more visible-sensingphotodetectors than infrared-sensing photodetectors.

For example, each infrared-sensing photodetector might be more sensitiveto the infrared projected pattern, while each visible-sensingphotodetector might be comparatively less sensitive to ambient light inthe environment. Thus, certain optical sensors might be designed toinclude more of one type of photodetector compared to the other,depending upon its intended use. Furthermore, the manner in which thevisible-sensing photodetectors and infrared-sensing photodetectors arearranged may vary depending upon the number of each type ofphotodetector and the intended purpose of the optical sensor. It shouldbe understood that various photodetectors that combine any number ofvisible-sensing photodetectors and infrared-sensing photodetectorsarranged in a variety of ways may be implemented, depending upon theimplementation, intended use, and/or the kinds of environments that therobotic system is expected to encounter.

Note that, although 16 photodetectors are depicted in a 4-by-4 array, anoptical sensor of the present application may include any number ofphotodetectors arranged a variety of dimensions. Also note that the term“mosaic” as referred to herein refers to a repeated grouping ofphotodetectors.

In some embodiments, the visible-sensing photodetectors may includephotodetectors for separately capturing red, green, and blue (RGB)visible light. FIG. 5B is a conceptual illustration of an arrangement510 of red-sensing, green-sensing, blue-sensing, and infrared-sensingphotodetectors. Unlike the example optical sensor shown in FIG. 5A,where the visible-sensing photodetectors capture all visible lightacross a wide band, the visible-sensing photodetectors shown in FIG. 5Bcapture visible light in smaller, separate bands for red, green, andblue visible light.

In some embodiments, it may be desired to capture color images of anenvironment in addition to performing stereoscopic depth sensing. Anoptical sensor with photodetectors arranged like those in FIG. 5B may beemployed in order to provide such color images. The control circuitry ofsuch an optical sensor may combine light detected from the RGBphotodetectors in order to generate a color image. In some cases, colorimages may enhance feature detection from visible light images byenabling the detection of subtle color changes that might be difficultto discern from panchromatic images.

It should be understood that “visible-sensing photodetectors” mightherein refer to either wide-band visible light photodetectors thatcapture panchromatic images from all colors of visible light or acombination of photodetectors that detect smaller bands within thevisible light spectrum (e.g. red light, green light, and blue light) forcapturing color images.

FIG. 6 is a conceptual illustration of an example stereo imaging system600 that includes a texture projector 602, a first optical sensor 604,and a second optical sensor 606. In FIG. 6, the first optical sensor 604is shown separated from the texture projector 602 by a distance D1 andthe second optical sensor 606 is shown separated from the textureprojector 602 by a distance D2. The first optical sensor 604 is shownseparated from the second optical sensor 606 by a distance of D3. In thestereo imaging system 600 of FIG. 6, the first optical sensor 604 andthe second optical sensor 606 may observe the environment 608 fromdifferent angles (e.g. from different perspectives). Thus, the firstoptical sensor 604 may be used to capture images of the environment 608from a first viewpoint, while the second optical sensor 606 may be usedto capture images of the environment 608 from a second viewpoint.

The first optical sensor 604 and the second optical sensor 606 may beconfigured to capture both visible light images and infrared lightimages. In some embodiments, first optical sensor 604 and the secondoptical sensor 606 are similar to or the same as the optical sensor 402,and may have photodetectors arranged in a manner described above(including the arrangement 500 shown in FIG. 5).

The environment 608 may be at least partially illuminated by visiblelight 610. This visible light source may be sunlight or visible-spectrumartificial lighting (e.g. from incandescent light bulbs, fluorescentlight bulbs, light-emitting diodes, etc.), among other possible visiblelight sources. The visible light 610 may brightly illuminate certainportions, objects, or surfaces within the environment 608, while otherportions, objects, or surfaces within the environment 608 are shadowedor dimly lit.

The environment 608 may also be partially illuminated by an infraredprojected pattern from the texture projector 602. The texture projector602 may be an infrared light emitter or infrared projector. Forinstance, the texture projector may include a light source, projectionoptics, and a liquid crystal display (LCD) or other method ofinterrupting light with a pattern. The projected pattern may be a knowncombination of grids, dots, stripes, spots, horizontal bars, and/orvertical bars, among other shapes.

In some instances, the projected pattern may only be detectable whenreflected from the portions, objects, or surfaces within the environment608 that are shadowed or dimly lit. Certain visible light sources may,in addition to emitting visible light, emit infrared light that washesout or overcomes the infrared projected pattern.

In some embodiments, the projected pattern may be a known or otherwisepredetermined pattern that is unique combination of shapes. When thepattern is predetermined, unique portions of that pattern may be locatedwithin captured infrared images and provide a reference point fortriangulation. For example, once a unique portion of a predeterminedpattern is identified from a captured image, properties of that uniqueportion of the predetermined pattern (such as its size and location) canbe used as a basis to determine the depth of a surface on which thatunique portion is projected. In implementations employed a known orpredetermined projected pattern, structured-light depth sensing mayrequire only one infrared image and the relative positions of theoptical sensor and texture projector in order to generate a depth map.However, multiple infrared light images captured from differentviewpoints may provide additional information that can be used to refineor verify the accuracy of the triangulation.

In other embodiments, the projected pattern may be randomly generated.In some implementations, multiple infrared light images may be capturedof a changing randomly-generated projected pattern. A computing deviceemploying structured-light processing may derive depth information foreach infrared light image. The depth information corresponding to eachdifferent pattern can then be combined to generate a more accurate orcomplete depth map.

During operation, the first optical sensor 604 may capture a visiblelight image from visible light 610 reflected off the environment 608 andan infrared light image from the infrared projected pattern reflectedoff the environment 608. From a different viewpoint, the second opticalsensor 606 may capture a visible light image from visible light 610reflected off the environment 608 and an infrared light image from theinfrared projected pattern reflected off the environment 608. The darkportions of the environment may appear to be black or very dark,depending upon the exposure time of the optical sensors, while thebrightly-lit portions of the environment 608 may reveal detail andfeatures of the illuminated surfaces in the visible light images.Conversely, the brightly-lit portions of the environment 608 may appearto white or washed out, depending upon the exposure time of the opticalsensors, while the dark portions of the environment 608 may reveal theprojected infrared pattern in the infrared light images.

A computing device may generate a depth image based on the two visiblelight images captured by the first optical sensor 604 and the secondoptical sensor 606. This may involve finding mappings of correspondingpixel values (e.g. pixel values of a particular feature) within the twovisible light images and calculating how far apart these common areasreside in pixel space and using triangulation (based on, for example,the distance D3 between the two optical sensors) to determine a depthmap or image. This depth map or depth image may contain informationrelating to the distances of surfaces of objects in the environment 608based on features detected from the reflected visible light. However,surfaces of objects in shadows or dimly lit portions of the environment608 may not reflect enough visible light in order for a distances ofthose surfaces to be accurately determined; thus, the depth map producedfrom the visible light images may accurately represent depths for someof the surfaces in the environment 608.

Similarly, the computing device may generate a depth image based on thetwo infrared light images captured by the first optical sensor 604 andthe second optical sensor 606. The depth image or depth map may begenerated for the infrared light images in a similar manner to that forthe visible light images. This depth map or depth image may containinformation relating to the distances of surfaces of objects in theenvironment 608 based on features detected from the reflected infraredlight (including the infrared projected pattern). However, surfaces ofobjects in brightly illuminated portions of the environment 608 mayreflect too much infrared light, thereby washing out otherwisedetectable features and making the determination of distances of thosesurfaces difficult; thus, the depth map produced from the infrared lightimages may accurately represent depths for some of the surfaces in theenvironment 608.

The computing device may combine information from the depth mapgenerated from the visible light images and the depth map generated fromthe infrared light images to determine a combined depth map of theenvironment 608. Because each depth map represents depths detected indifferent light spectra, combining the depth maps may provide a morecomprehensive and/or accurate depth map for the whole environment 608.

In some embodiments, a depth map may be determined based on a singleinfrared light image captured by either the first optical sensor 604 orthe second optical sensor 606. For example, a computing device mayemploy triangulation techniques based on the known distance between thetexture project 602 and the first optical sensor 604 and the capturedinfrared image of the projected pattern onto environment 608 by thefirst optical sensor 604 to determine the depth map. Any combination ofknown distances D1, D2, and D3 and one or more infrared images capturedby the first optical sensor 604 and/or the second optical sensor 606 maybe used to determine a depth map.

In some instances, a depth map determined from visible light images andanother depth map determined from infrared light images might containoverlapping depth information. When generating the combined depth map,the computing device may select the depth information from one of thetwo depth maps to use in the combined depth map. In someimplementations, the computing device may average the depth informationfrom both depth maps where overlapping depth information exists.

Although the depth map determined from visible light images requires atleast two visible light images captured from two different viewpoints,the depth map determined from the infrared light images may only requirea single infrared light image captured from one of those viewpoints. Asdescribed above, the infrared-based depth map may be determined usingtriangulation based on the known distance between the texture projector602 and the optical sensor capturing the infrared light image. Thus, inother example stereo imaging systems, only one of the first opticalsensor 604 and the second optical sensor 606 requires the capability tocapture infrared images in order to carry out the combined depth maptechniques from based on visible light and infrared light of the presentapplication.

FIG. 7 is a block diagram of an example method 700 for determining avirtual representation of an environment. Method 700 shown in FIG. 7presents an embodiment of a method that could be used or implemented bythe robotic device 200 of FIG. 2B, for example, or more generally by oneor more components of any computing device. Method 700 may include oneor more operations, functions, or actions as illustrated by one or moreof blocks 702-714. Although the blocks are illustrated in a sequentialorder, these blocks may also be performed in parallel, and/or in adifferent order than those described herein. Also, the various blocksmay be combined into fewer blocks, divided into additional blocks,and/or removed based upon the desired implementation.

In addition, for the method 700 and other processes and methodsdisclosed herein, the block diagram shows functionality and operation ofone possible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor orcomputing device for implementing specific logical functions or steps inthe process. The program code may be stored on any type ofcomputer-readable medium, for example, such as a storage deviceincluding a disk or hard drive. The computer-readable medium may includenon-transitory computer-readable medium, for example, such ascomputer-readable media that stores data for short periods of time likeregister memory, processor cache and random access memory (RAM). Thecomputer-readable medium may also include non-transitory media, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer-readable media may also be any other volatile ornon-volatile storage systems. The computer-readable medium may beconsidered a computer-readable storage medium, for example, or atangible storage device.

In addition, for the method 700 and other processes and methodsdisclosed herein, each block in FIG. 7 may represent circuitry that iswired to perform the specific logical functions in the process.

In one embodiment, functions of the method 700 may be performed by acontrol system, such as 700 control system 240 of FIG. 2B. In otherembodiments, the functions of the method 700 may be distributed acrossmultiple control systems that are configured to use an output of stereoimage processing to determine the depths of surfaces in an environment.

Blocks 702, 704, and 706 are associated with depth perception fromvisible light images. Blocks 708, 710, and 712 are associated with depthperception from infrared light images. Although these two groups ofblocks are depicted as being performed in parallel, each of the blocks702-714 of method 700 shown in FIG. 7 may be performed in any order. Thearrangement of the operations of method 700 is intended to facilitateunderstanding of the method 700, and should not be misconstrued aslimiting the order in which the operations of blocks 702-714 may beperformed.

At block 702, the method 700 involves receiving, from a first pluralityof photodetectors of a first optical sensor, a first visible light imageof an environment as perceived from a first viewpoint. The first opticalsensor may include a combination of visible-sensing photodetectors andinfrared-sensing photodetectors, similarly to the optical sensor 402described above with respect to FIG. 4. The photodetectors may bearranged in a variety of ways, such as the checkerboard pattern asdescribed with respect to FIG. 5, among other possible arrangements. Thefirst optical sensor may also be similar to the first optical sensor 604shown in FIG. 6.

Light within the visible spectrum may cause the first plurality ofphotodetectors to generate a charge. The charge generated by aparticular photodetector may be proportional to the intensity orluminance of light incident on that particular photodetector. The firstoptical sensor may include control circuitry configured to read thecharges generated by the first plurality of photodetectors and, based onthose charges, generate an image. In some embodiments, the image ispanchromatic (e.g. grayscale) within the human-visible wavelengthspectrum. In other embodiments, the first plurality of photodetectorsincludes photodetectors that separately capture red, green, and bluevisible light, and the control circuitry may be configured to combine(e.g. demosaic) those charges to form a color image.

At block 704, the method 700 involves receiving, from a first pluralityof photodetectors of a second optical sensor, a second visible lightimage of the environment as perceived from a second viewpoint. The firstoptical sensor may be positioned a known distance from the secondoptical sensor. In some embodiments, the relative angles of the firstand second optical sensors are also known. A computing device may usethe respective distance and angles between the first and second opticalsensors to estimate the depth of at least one surface in theenvironment.

At block 706, the method 700 involves identifying first correspondingfeatures of the environment between the first visible light image andthe second visible light image. The environment may have therein anumber of objects that form surfaces within the environment. Thoseobjects may have certain visually-recognizable features—such as edges,changes in color, and contours, among other possible features—that canbe commonly recognized in both the first and second visible lightimages. A computing device may identify a given feature of theenvironment and determine a first pixel location of that feature withinthe first visible light image and a second pixel location of thatfeature within the second visible light images. Based on differencebetween the first and second pixel location and the relative positionsof the first and second optical sensors, the depth of that feature (e.g.the distance between the optical sensors and the feature) may bedetermined using triangulation and/or stereo image processing. Suchfeature depth determinations may be repeated for any number ofidentified features within the environment.

At block 708, the method 700 involves receiving, from a second pluralityof photodetectors of a first optical sensor, a first infrared lightimage of an environment as perceived from a first viewpoint. Theinfrared light described in the method 700 may be light within aparticular band of infrared light (e.g. 825 nm to 875 nm, among otherpossible IR bands). The first infrared light image of the environmentmay capture residual infrared light within that particular band fromvarious light sources. In some embodiments, the first infrared lightimage may also capture an infrared pattern projected onto theenvironment by a texture projector.

At block 710, the method 700 involves receiving, from a second pluralityof photodetectors of a second optical sensor, a second infrared lightimage of an environment as perceived from a second viewpoint. Like thefirst infrared light image, the second infrared light image may capturean infrared pattern projected onto the environment by a textureprojector.

At block 712, the method 700 involves identifying second correspondingfeatures of the environment between the first infrared light image andthe second infrared light image. Residual infrared light within theparticular infrared band might provide adequate illumination to detectcertain features of objects or surfaces within the environment.Additionally, an infrared pattern projected onto the environmentproduces detectable features on otherwise dimly-lit or featurelesssurfaces in the environment. As described with respect to block 706, acomputing device may identify features (naturally existing and/orartificially projected feature) in the environment, determine therelative pixel locations of those identified features between the twoinfrared light images, and determine the depths of those features usingtriangulation and/or stereo image processing.

At block 714, the method 700 involves determining a depth estimate forat least one surface in the environment based on the first correspondingfeatures and the second corresponding features. In the context of asingle surface, a computing device may determine a first depth estimatefor that surface based on the first corresponding features and a seconddepth estimate for that surface based on the second correspondingfeatures. The computing device may employ stereo image processing and/ortriangulation techniques in order to determine the first and seconddepth estimates. In some embodiments, the computing device may determinea combined depth estimate by averaging the first and second depthestimates.

A particular feature of a given surface may only be identified in eitherthe visible light images or the infrared light images. In such cases,determining the depth estimate may involve determining the depthestimate based on the pair of images from which that particular featuremay be identified.

In the context of the whole environment (or a portion of the environmentcontaining multiple surfaces), a computing device may generate a visiblelight-based depth map for various surfaces in the environment based onidentified corresponding features between the visible light images andan infrared light-based depth map for various surfaces in theenvironment based on identified corresponding features between theinfrared light images. The visible light-based depth map may lack depthinformation for dimly-lit portions of the environment, while theinfrared light-based depth map may lack depth information forbrightly-lit portions of the environment. In some implementations, thecomputing device may determine a combined depth map by selecting one ofthe two depth maps as a “base” depth map, then including depthinformation missing from the selected depth map from the other depthmap.

In other implementations, the computing device may determine a combineddepth map by including unique depth information from either depth mapand, where depth information for a given surface exists in both depthmaps, averaging the depth information from both depth maps for thatgiven surface. For example, the computing device may determine whichdepth information is unique to the visible light-based depth map, whichdepth information is unique to the infrared light-based depth map, andwhich depth information exists in both depth maps. First, the computingdevice may stitch or otherwise join together the unique depthinformation from each depth map. Then, the computing device may averagethe depth information for surfaces whose depths are present in bothdepth maps and stitch or otherwise combine that averaged depthinformation with the unique depth information to form a combined depthmap.

Note that “depth map” and “depth image” may refer to spatial depthinformation of surfaces in an environment. Example representations ofdepth maps are depicted in FIG. 11B and FIG. 11D below.

In some instances, the computing device may repeat one or more blocks ofthe method 700 to determine multiple depth estimates for an environment.For example, the computing device may use the method 700 to determine adepth map that indicates the depth to one or more surfaces in theenvironment.

In various embodiments, determining a depth map or depth image mayinvolve identifying a feature from an image from one perspective andlocating that same feature within another image from a differentperspective. The relative pixel location of that feature between the twoimages may be used as a basis for determining the depth (e.g. distancefrom the optical sensor) by triangulation. In various implementations,the identifying these corresponding features may involve determining oneor more correlation surfaces. A “correlation surface” as describedherein refers to a combination of numerical estimations indicative of anextent of similarity between a pixel and its neighborhood (e.g. otherpixels adjacent or otherwise nearby that pixel) within one image and twoor more pixels (e.g. pixels and their neighborhoods) within a differentimage. FIG. 8A, FIG. 8B, and FIG. 8C illustrate the manner in whichcorrelation surfaces may be used in various implementations in theidentification of corresponding features between two images. In variousinstances, when determining a correlation surface, a given pixel (suchas pixel X_(L)) and its “neighborhood” in the left image may be comparedto one or more pixels in the right image. Considering the neighborhoodsurrounding a given pixel X_(L) when determining a correlation surfacemay be performed in the present example. Additionally, it should beunderstood that the process may be flipped, such that a pixel in theright image may be compared to two or more pixels in the left image.

FIG. 8A is a conceptual illustration 800 of an example configuration fordetermining a correlation surface, according to an example embodiment.The configuration includes a left optical sensor 802 and a right opticalsensor 804 arranged to provide different viewpoints of an environment.The following example discussed with respect to FIG. 8A, FIG. 8B, andFIG. 8C describe a process by which a correlation surface can bedetermined for pixel X_(L). Pixel X_(L) of an image captured by the leftoptical sensor 802 may be a pixel within the left viewpoint image havingan x-value and y-value coordinate position within the image.

When determining the correlation surface for pixel X_(L), a computingdevice may compare aspects of the pixel X_(L) (e.g. brightness,luminosity, color, etc.) in the left image to aspects of one or morepixels in the right image, such as pixels X_(R), (X+1)_(R), (X+2)_(R),(X−1)_(R), and (X−2)_(R). The computing device may determine an extentof similarity between the left pixel X_(L) and one or more of thoseright image pixels. The numerical value representing the extent ofsimilarity between two pixels may be referred to herein as a“correlation value,” such that a set of correlation values serve as thebasis of a correlation surface. In the present example, smallercorrelation values are indicative of a higher degree of similaritybetween two pixels, whereas larger correlation values are indicative ofa lesser degree of similarity between two pixels. Thus, any “dips,”“valleys,” or localized minima may be interpreted as two pixels havingthe highest relative degree of similarity compared to the othersurrounding pixels, which in some cases may signify a “match” (e.g.corresponding features).

As an example operation, a computing device determines the correlationsurface for X_(L) by first comparing pixel X_(L) to each of the pixelsX_(R), (X+1)_(R), (X+2)_(R), (X−1)_(R), and (X−2)_(R). The correlationsurface 810 may be generated after performing those comparisons. Asshown in FIG. 8B, the correlation surface 810 contains a localizedminimum at point p₄ corresponding to pixel (X+1)_(R), as indicated bythe “valley” 812 in the correlation surface 810. In this example, thecomputing system may thus determine that the feature in X_(L)corresponds to the feature in (X+1)_(R). Upon identifying the relativepixel locations for the corresponding feature, the computing device mayuse triangulation to determine the depth of the feature in X_(L) asbeing equal to the distance from the left optical sensor 802 to pointp₄. Such a triangulation technique may involve determining anintersection point of projection line 806 of X_(L) and a projection line808 of (X+1)_(R). The projection line 806 for X_(L) may be determinedbased on focal point F_(L) of the left optical sensor and the projectionline 808 for (X+1)_(R) may be determined based on focal point F_(R) ofthe right optical sensor.

Note that, in this example, as the “x” pixel on the right imageincreases (e.g. “x” pixels that are further to the right), the depth ofthe corresponding point p decreases; conversely, as the “x” pixel on theright image decreases (e.g. “x” pixels that are further to the left),the depth of the corresponding point p increases. Thus, afteridentifying the relative pixel locations of corresponding features fromthe correlation surface, a computing device may also estimate the depthof that feature.

Also note that the example described above with respect to FIG. 8A andFIG. 8B discusses a process of calculating a correlation surface forleft image pixel X_(L) to a set of right image pixels that arehorizontally adjacent to each other. However, various embodiments mayinvolve calculating a correlation surface by comparing the left imagepixel X_(L) to a set of right image pixels that are vertically adjacentto each other, a set of right image pixels that make up atwo-dimensional array of pixels, or a set of right image pixels that arewithin a predetermined radius of a given right image pixel, among otherpossible sets of right image pixels. The manner in which the correlationsurface is determined for a given left image pixel may depend upon avariety of factors, including the relative positions of the two opticalsensors in a given system or the processing speed of a computing deviceused to determine the correlation system, among other possible factors.

The correlation surface as described herein may refer to a collection ofdiscrete correlation values. Any curve-fitting or continuousrepresentations of the correlation surface are provided for explanatorypurposes; various embodiments may or may not determine a functionalexpression (e.g. a polynomial equation or other interpolated function).

In some embodiments, multiple correlation surfaces may be determinedcorresponding to images captured from different wavelengths of light.For example, two optical sensors may capture a pair of visible lightimages and a pair of infrared light images. In an example process, acomputing device may determine one correlation surface from the pair ofvisible light images, and another correlation surface from the pair ofinfrared light images. Then, the computing device may combine the twocorrelation surfaces to determine a combined correlation surface. Anexample illustration of this example process is shown in FIG. 8C.

In an example scenario, a feature may only be detectable within onewavelength band of light. For example, certain features of surfaces indark portions of an environment may be too dark for a visible lightimage to determine distinguishable features therefrom. However, aninfrared light projector may illuminate the otherwise dark surface andallow features to be detected in an infrared light spectrum.

In FIG. 8C, the correlation surface 830 for pixel X_(L) determined fromvisible light images does not contain any dips, valleys, or localizedminima. However, the correlation surface 840 for pixel X_(L) determinedfrom infrared light images has a localized minimum at point p₄. Acomputing device may add correlation surface 830 and correlation surface840 to determine a combined correlation surface 850. By adding thecorresponding p values between the two correlation surfaces 830 and 840,the localized minima is maintained in correlation surface 850. Thus, bysimply adding two correlation surfaces together that were eachdetermined from different light wavelengths, the resulting combinedcorrelation surface 850 may preserve the local minima from both of theseparate correlation surfaces. Then, the computing device may identifythese local minima from the combined correlation surface 850, identifywhich of those localized minima represent corresponding features with agiven pixel value, and determine the depth of the feature in that givenpixel value as described above.

Identifying localized minima may be achieved using a variety oftechniques. In some implementations, a computing device may determine ifa correlation value in a given correlation surface is less than athreshold correlation value. In other implementations, a computingdevice may determine if the smallest correlation value in a givencorrelation surface is less than the next smallest correlation value. Avariety of other techniques may be used to identify a localized minimumin a correlation surface.

FIG. 9 is a conceptual illustration 900 of combining information frommultiple depth images to form a combined output depth image. A computingdevice may employ stereo image processing on visible light image A andvisible light image B to determine a first depth image 902. The firstdepth image may be a mapping of distances from one or more opticalsensors to at least one surface in an environment. The computing devicemay also employ stereo image processing on IR light image A and IR lightimage B to determine a second depth image 904. Like the first depthimage, the second depth image may be a mapping of distances from one ormore optical sensors to at least one surface in the environment. Thecomputing device may combine the first depth image 902 and the seconddepth image 904 in order to determine the output depth image 906.

FIG. 10A is a conceptual illustration 1000 of an example environment,according to an example embodiment. The example environment includeswall 1008 and box 1010 positioned in front of the wall with respect totexture projector 1002. Texture projector 1002 may be placed at adistance from the wall 1008 and box 1010, and configured to project aninfrared projected pattern, similarly to or the same as textureprojector 602 as described with respect to FIG. 6. Additionally, a firstoptical sensor 1004 and a second optical sensor 1006 may be present andcapture both visible light images and infrared light images of theexample environment. The texture projector 1002, first optical sensor1004, and second optical sensor 1006 may be configured in a similarmanner as the texture projector 602, first optical sensor 604, andsecond optical sensor 606 as described with respect to FIG. 6.

The example environment shown in FIG. 10A is partially lit by a brightlight source, such that the environment has a bright area 1014(illustrated using a dotted pattern) and a dark area 1012. The brightarea 1014 may be illuminated by an artificial light source (e.g. a lightbulb) and/or by sunlight. The dark area 1012 may be produced by a shadowcast from an object onto the environment, or be a result of a lack of alight source illuminating that portion of the environment. Both the wall1008 and the box 1010 include dark and bright portions, as illustratedin FIG. 10A. The texture projector 1002 may project the pattern onto theentire environment; however, an optical sensor may only perceive theinfrared projected pattern on the dark area 1012. Thus, the dark area1012 is depicted having the infrared pattern projected thereon.

During operation, the optical sensors 1004 and 1006 capture both visiblelight images and infrared light images from different viewpoints. Theprocess of determining depth information from these captured images isdepicted in FIG. 10B and described below.

Note that the illustration 1000 depicts bright area using dots and thedark area as having a visibly perceivable projected pattern. Theseillustrated patterns are provided for explanatory purposes and do notnecessarily correspond to an optical sensor's perception or a humaneye's perception of the environment. Also note that, although theboundary between the dark area 1012 and the bright area 1014 is distinctin the illustration 1000 of FIG. 10A, the boundary between dark andbright areas in other environments may be more of a gradient.

FIG. 10B is an example illustration of determining depth from multipledepth images. The left side of FIG. 10B depicts determining a depthestimate from the dark area illuminated by the infrared projectedpattern. The shaded area on the illustration 1020 of the environmentrepresents the portion of the environment on which the projected patterncan be detected by infrared light images and used to determine thedepths of the surfaces in the environment. For the purposes ofexplanation, estimating the depth of the surfaces across horizontalsection 1022 is discussed. Using structured-light processing, acomputing device may determine depth information from the infrared lightimages of the dark area of the environment illuminated by the infraredprojected pattern. The resulting depth estimate is depicted in graph1024. Moving left to right, the depth of the far wall is detected, andthen the depth decreases as the box is detected until the dark areastops (represented by the dot). Note that, in this example, no depthinformation is determined for the bright area based on the infraredlight images; this is because the bright light overcame the infraredprojected pattern, thereby making the discernment of features on thebright area difficult.

The right side of FIG. 10B depicts determining a depth estimate from thebright area. The shaded area on the illustration 1030 of the environmentrepresents the portion of the environment illuminated by the brightlight, which allows for visual detection of features from the visiblelight images and used to determine depths of the surfaces in theenvironment. Using stereo image processing, a computing device maydetermine depth information from the visible light images of the brightarea of the environment. The resulting depth estimate is depicted ingraph 1034. Moving left to right, no depth information is determinedalong the dark area until the bright area is reached. This is because,in the visible light spectrum, the dark area does not provide sufficientlight to discern features used to sense depth. At that boundary betweenthe bright and dark area (represented by the dot in graph 1034), thedepth of the box is detected, and then the depth of the wall.

Graph 1040 depicts the combined depth estimate from the depth estimatesshown in graph 1024 and graph 1034. Although the environment had a wideluminance range, employing the depth sensing techniques of the presentapplication produced a depth estimate of both the bright and dark areasof the environment. As illustrated in FIGS. 10A and 9B, a given surfacemight have both bright and dark portions that, using typical stereoimaging techniques, would provide insufficient or inaccurate depthinformation about that surface. However, by detecting light in theenvironment in the visible spectrum and a projected pattern onto theenvironment within a particular infrared band, comprehensive andaccurate depth information of surfaces in the environment may bedetected. The techniques of the present application may provide improveddepth sensing in sunlit environments, darker indoor environments, and acombination of light and dark environments.

FIG. 11A is an example image 1100 of a scene, according to an exampleembodiment. The scene shown in FIG. 11A includes an opened garage or baydoor through which sunlight illuminates a portion of the ground. Otherportions of the ground and interior walls are dimly lit in comparison tothe luminance of the sunlight. Typical stereo image processing mightproduce a depth map 1110 of the scene as shown in FIG. 11B. In FIG. 11B,black portions of the environment indicate a lack of depth information,while the white or gray portions of the environment indicate thatdetermined depth information (where light gray indicates a smaller depthand dark gray indicates greater depth). As shown in FIG. 11B, depthinformation could not be determined for most of the shadowed portions ofthe scene.

FIG. 11C is an example image 1120 of the scene with a projected pattern,according to an example embodiment. The image 1120 shows both thesunlight illuminating the ground and a projected pattern illuminatingthe shadowed portions of the environment. Note that image 1120 mayrepresent a broad range of light wavelengths, and may simultaneouslydepict captured visible light and/or infrared light for the purposes ofexplanation. By performing the combined infrared and visible light depthsensing of the present application, a computing device may generatedepth map 1130 of the scene as shown in FIG. 11D. Unlike the depth map1110 shown in FIG. 11B, the depth map 1130 shown in FIG. 11D includesdepth information from the shadowed portion of the scene (which may havebeen derived from infrared light images using structured-lightprocessing). Compared to depth map 1110, depth map 1130 determined fromthe combined visible light and infrared light depth sensing provides amore comprehensive depth information.

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

Since many modifications, variations, and changes in detail can be madeto the described examples, it is intended that all matters in thepreceding description and shown in the accompanying figures beinterpreted as illustrative and not in a limiting sense. Further, it isintended to be understood that the following further describe aspects ofthe present description.

What is claimed is:
 1. An optical sensor comprising: a planar array ofphotodetectors comprising a first plurality of photodetectors and asecond plurality of photodetectors, wherein each photodetector isconfigured to generate a charge based on an intensity of light incidenton the photodetector; a light filter array coupled to the planar arrayof photodetectors comprising a first plurality of filters configured topass visible light onto the first plurality of photodetectors and asecond plurality of filters configured to pass infrared light within aparticular infrared band onto the second plurality of photodetectors,wherein filters of the first plurality of filters are interspersed withfilters of the second plurality of filters; and a control circuitconfigured to: generate a visible light image based on charges from thefirst plurality of photodetectors; and generate an infrared light imagebased on charges from the second plurality of photodetectors.
 2. Theoptical sensor of claim 1, wherein the first plurality of filters andthe second plurality of filters are arranged in a checkerboard pattern.3. The optical sensor of claim 2, wherein the first plurality ofphotodetectors and the second plurality of photodetectors are arrangedin a checkerboard pattern that corresponds to the checkerboard patternof the first plurality of filters and the second plurality of filters.4. The optical sensor of claim 1, wherein the first plurality of filterscomprises a first subset of filters configured to pass red visiblelight, a second subset of filters configured to pass green visiblelight, and a third subset of filters configured to pass blue visiblelight.
 5. The optical sensor of claim 4, wherein the first plurality offilters and the second plurality of filters are arranged to form asquare tiling pattern comprising a plurality of filter mosaics, whereineach filter mosaic comprises (1) a filter of the first subset thatpasses red visible light, (2) a filter of the second subset that passesgreen visible light, (3) a filter of the third subset that passes redvisible light, and (4) a filter of the second plurality of filters thatpasses infrared light within the particular infrared band.
 6. Theoptical sensor of claim 1, wherein the control circuit is furtherconfigure to: cause, over a first duration of time, the first pluralityof photodetectors to capture visible light to generate the charges thatthe visible light image is based on; and cause, over a second durationof time, the second plurality of photodetectors to capture infraredlight to generate the charges that the infrared light image is based on,wherein the second duration of time is greater than the first durationof time.
 7. The optical sensor of claim 1, wherein the control circuitis further configured to: amplify the charges generated by the firstplurality of photodetectors by a first gain value; and amplify thecharges generated by the second plurality of photodetectors by a secondgain value, wherein the second gain value is greater than the first gainvalue.
 8. The optical sensor of claim 1, wherein a number ofphotodetectors in the first plurality of photodetectors is greater thana number of photodetectors in the second plurality of photodetectors. 9.The optical sensor of claim 1, wherein the control circuit is furtherconfigured to transmit the visible light image and the infrared lightimage to one or more devices.
 10. A system comprising: a planar array ofphotodetectors comprising a first plurality of photodetectors configuredto capture visible light and a second plurality of photodetectorsconfigured to capture infrared light within a particular infrared band,wherein each photodetector is configured to generate a charge based onan intensity of light incident on the photodetector, and wherein thefirst plurality of photodetectors and the second plurality ofphotodetectors alternate along both rows and columns to form acheckerboard pattern; a light filter array coupled to the planar arrayof photodetectors comprising a first plurality of filters configured topass visible light onto the first plurality of photodetectors and asecond plurality of filters configured to pass infrared light within aparticular infrared band onto the second plurality of photodetectors;and a computing device configured to: generate a visible light imagebased on charges from the first plurality of photodetectors; andgenerate an infrared light image based on charges from the secondplurality of photodetectors.
 11. The system of claim 10, wherein thefirst plurality of filters comprises a first subset of filtersconfigured to pass red visible light, a second subset of filtersconfigured to pass green visible light, and a third subset of filtersconfigured to pass blue visible light.
 12. The system of claim 11,wherein the first plurality of filters and the second plurality offilters are arranged to form a square tiling pattern comprising aplurality of filter mosaics, wherein each filter mosaic comprises (1) afilter of the first subset that passes red visible light, (2) a filterof the second subset that passes green visible light, (3) a filter ofthe third subset that passes red visible light, and (4) a filter of thesecond plurality of filters that passes infrared light within theparticular infrared band.
 13. The system of claim 10, wherein thecomputing device is further configure to: cause, over a first durationof time, the first plurality of photodetectors to capture visible lightto generate the charges that the visible light image is based on; andcause, over a second duration of time, the second plurality ofphotodetectors to capture infrared light to generate the charges thatthe infrared light image is based on, wherein the second duration oftime is greater than the first duration of time.
 14. The system of claim10, wherein the computing device is further configured to: amplify thecharges generated by the first plurality of photodetectors by a firstgain value; and amplify the charges generated by the second plurality ofphotodetectors by a second gain value, wherein the second gain value isgreater than the first gain value.
 15. The system of claim 10, whereinfilters of the first plurality of filters are interspersed with filtersof the second plurality of filters.
 16. A method comprising: receiving,at a computing system, a first set of charges generated based onrespective intensities of light incident on a first plurality ofphotodetectors; based on the first set of charges, generating a visiblelight image; receiving, at the computing system, a second set of chargesgenerated based on respective intensities of light incident on a secondplurality of photodetectors, wherein the first plurality ofphotodetectors and the second plurality of photodetectors alternatealong both rows and columns to form a checkerboard pattern; and based onthe second set of charges, generating an infrared light image.
 17. Themethod of claim 16, further comprising: causing, over a first durationof time, the first plurality of photodetectors to capture visible lightto generate the first set of charges; and causing, over a secondduration of time, the second plurality of photodetectors to captureinfrared light to generate the second set of charges, wherein the secondduration of time is greater than the first duration of time.
 18. Themethod of claim 16, further comprising: amplifying the first set ofcharges generated by the first plurality of photodetectors by a firstgain value; and amplifying the second set of charges generated by thesecond plurality of photodetectors by a second gain value, wherein thesecond gain value is greater than the first gain value.
 19. The methodof claim 16, wherein the first set of charges and the second set ofcharges are received simultaneously.
 20. The method of claim 16, whereinthe first set of charges are received at a first time and the second setof charges are received at a second time, wherein the second time issubsequent to the first time.